Expectation
Expectation
Example 1. Consider a probability space (\Omega, \sF, P). We consider N trials of a random experiment, and define a random vector X: \Omega \to \sX^N such that X_i \triangleq \pi_i\circ X :\Omega \to\sX is a discrete random variable associated with the trial i \in [N]. If the marginal distributions of random variables (X_1, \dots, X_N) are identical with the common probability mass function P_{X_1}: \sX \to [0,1], then the empirical mean of random variable X_1 can be written as For a random variable X_1: \Omega \to \sX, we can define events E_{X_1}(x) \triangleq X_1^{-1}\set{x} for each value x \in \sX. The probability mass function P_{X_1}:\sX \to [0,1] for the discrete random variable X_1 can be estimated for each x \in \sX as the empirical probability mass function
Recall that a simple random variable X_1 can be written as X_1 = \sum_{x \in \sX}x\Ind{E_{X_1}(x)}, where E_{X_1} \triangleq (E_{X_1}(x) \in \sF: x \in \sX) is a finite partition of the sample space \Omega and P_{X_1}(x) = P(E_{X_1}(x)). That is, we can write the empirical mean in terms of the empirical PMF as
This example motivates the following definition of mean for simple random variables.
Definition 2 (Expectation of simple random variable). The mean or expectation of a simple random variable X:\Omega \to \sX \subseteq \R defined on a probability space (\Omega, \sF, P), is denoted by \E[X] and defined as
Remark 1. For an indicator random variable \Ind{A}, we have \E\Ind{A} = P(A). That is, the expectation of an indicator function is the probability of the indicated set.
Remark 2. Since a simple random variable can be written as X = \sum_{x \in \sX}x\Ind{E_X(x)} where E_X(x) \triangleq X^{-1}\set{x} for all x\in\sX, we can write the expectation of a simple random variable as an integral
Theorem 3. Consider a non-negative random variable X: \Omega \to \R_+ defined on a probability space (\Omega, \sF, P). There exists a sequence of non-decreasing non-negative simple random variables Y: \Omega \to \R_+^\N such that for all \omega \in \Omega Then \E[Y_n] is defined for each n \in \N, the sequence (\E[Y_n] \in \R_+: n \in \N) is non-decreasing, and the limit \lim_n\E[Y_n] \in \R_+\cup\set{\infty} exists. This limit is independent of the choice of the sequence and depends only on the probability space.
Proof. Proof. For each n \in \N and k \in \set{0, \dots, 2^{2n}-1}, we define half-open sets B_{n,k} \triangleq (k2^{-n}, (k+1)2^{-n}]. Then, the collection of sets B_n \triangleq (B_{n,k}: k \in \set{0, \dots, 2^{2n}-1}) partitions the set (0, 2^n] for each n \in \N. Further, we observe that \cup_{n \in \N}(0, 2^n] = \R^+ and that B_{n+1,2k}\cup B_{n+1,2k+1} = B_{n,k} for all n \in \N and k.
For a non-negative random variable X: \Omega \to \R_+, we define events A^X_{n,k} = X^{-1}(B_{n,k}) \in \sF, and a sequence of simple non-negative random variables Y: \Omega\to\R_+^\N in the following fashion We observe that Y_n is a quantized version of X, and its value is the left end-point k2^{-n} when X \in B_{n,k} for each k \in \set{0, \dots, 2^{2n}-1}. Since \cup_{k=0}^{2^{2n}-1}A^X_{n,k} = X^{-1}(0, 2^n], it follows that we cover the positive real line as n grows larger and the step size grows smaller. Thus, the limiting random variable can take all possible non-negative real values. We observe that We see that Y_n(\omega) \le Y_{n+1}(\omega) \le X(\omega) and \lim_nY_n(\omega) = X(\omega) for all \omega \in \Omega.
Since Y_n: \Omega \to \R is a simple random variable for all n \in \N, the expectation \E[Y_n] is defined for all n, and can be written as We observe that this expectation is completely specified by the distribution function F_X, and we can write the limit ◻
Definition 4 (Expectation of a non-negative random variable). For a non-negative random variable X: \Omega \to \R defined on the probability space (\Omega,\sF,P), consider the sequence of non-decreasing simple random variables Y: \Omega \to \R_+^\N such that \lim_nY_n = X. The expectation of the non-negative random variable X is defined as
Remark 3. From the definition, it follows that \E[X] = \int_{\R_+} x dF_X(x).
Definition 5 (Expectation of a real random variable). For a real-valued random variable X defined on a probability space (\Omega, \sF, P), we can define the following functions
2 &X_+ , & &X_- .
We can verify that X_+, X_- are non-negative random variables and hence their expectations are well defined. We observe that X(\omega) = X_+(\omega) - X_-(\omega) for each \omega \in \Omega. If at least one of the \E[X_+] and \E[X_-] is finite, then the expectation of the random variable X is defined as
Theorem 6 (Expectation as an integral with respect to the distribution function). For a random variable X:\Omega\to\R defined on the probability space (\Omega, \sF, P), the expectation is given by
Proof. Proof. It suffices to show this for a non-negative random variable X, and the result follows from the definition of expectation of a non-negative random variable as the limit of expectation of approximating simple functions. ◻
Properties of Expectations
Theorem 7 (Properties). Let X: \Omega \to \R be a random variable defined on the probability space (\Omega, \sF, P).
Linearity: Let a, b \in \R and X, Y be random variables defined on the probability space (\Omega, \sF, P). If \E X, \E Y, and a\E X + b\E Y are well defined, then \E(aX+bY) is well defined and
Monotonicity: If P\set{X \ge Y} = 1 and \E[Y] is well defined with \E[Y] > -\infty, then \E[X] is well defined and \E[X] \ge \E[Y].
Functions of random variables: Let g: \R \to \R be a Borel measurable function, then g(X) is a random variable with \E[g(X)] = \int_{x\in \R}g(x)dF(x).
Continuous random variables: Let f_X: \R \to [0, \infty) be the density function, then \E X = \int_{x \in \R}xf_X(x)dx.
Discrete random variables: Let P_X: \sX \to [0,1] be the probability mass function, then \E X = \sum_{x \in \sX}xP_X(x).
Integration by parts: The expectation \E X = \int_{x \ge 0}(1-F_X(x))dx - \int_{x < 0}F_X(x)dx is well defined when at least one of the two parts is finite on the right hand side.
Proof. Proof. It suffices to show properties (i)-(iii) for simple random variables.
Let X = \sum_{x \in \sX}x\mathbbm{1}_{E_X(x)} and Y = \sum_{y \in \sY}y\mathbbm{1}_{E_Y(y)} be simple random variables, then (E_X(x)\cap E_Y(y) \in \sF: (x, y)\in \sX \times \sY) partition the sample space \Omega. Hence, we can write aX + bY = \sum_{(x,y) \in \sX \times \sY}(ax+by)\SetIn{E_X(x)\cap E_Y(y)} and from linearity of sum it follows that
From the fact that X-Y \ge 0 almost surely and linearity of expectation, it suffices to show that \E X \ge 0 for non-negative random variable X. It can easily be shown for simple non-negative random variables, and follows for general non-negative random variables by taking limits.
It suffices to show this holds true for simple random variables X: \Omega \to \sX \subset \R. Since g: \R \to \R is Borel measurable, Y \triangleq g(X): \Omega\to\sY \triangleq g(\sX) is a random variable. It follows that, we can write the following disjoint union \sX = \cup_{y\in \sY}\cup_{x \in g^{-1}\set{y}}\set{x}. Further, for each y \in \sY, we have Since E_X(x) are disjoint for all x \in \sX, we get P(E_Y(y)) = \sum_{x \in g^{-1}\set{y}}P(E_X(x)). Using the above two facts, we can write the expectation
For continuous random variables, we have dF_X(x) = f_X(x)dx for all x \in \R.
For discrete random variables X : \Omega \to \sX, we have dF_X(x) = P_X(x) for all x \in \sX and zero otherwise.
We can write \E X = \int_{x <0}xdF_X(x)-\int_{x \ge 0}xd(1-F_X)(x). We apply integration by parts to the first term on the right, to get Similarly, we apply integration by parts to the second term on the right, to get
◻